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ABSTRACT 



This paper presents the characteristic features of a method 
for description and evaluation of learning environments which aims at the 
improvement of the traditional questionnaire-based methods and the 
integration of the varying points of view of design, support for learning, 
and evaluation. This method, called Integrated Pedagogical Profile (IPP) can 
be seen as an advancement of the profile of pedagogical dimensions presented 
by Thomas Reeves, but with some significant reconstructions. The solution 
presented in this paper is based on the reconstruction of the dimensions and 
the creation of a fuzzy picture that can give an overall insight into the 
many aspects of the learning environment, yet without giving a false 
impression of preciseness. This dynamic picture, which integrates the above 
mentioned points of view in interdisciplinary fashion, is achieved by 
gathering the individual results and representing them by circles, the 
opacity of which is related to the size of the sample, this forming a general 
picture of the essential features. (Contains 14 references.) (Author/AEF) 
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Abstract: In this paper we will present the characteristic features of a method for description and 
evaluation of learning environments which pursues to the improvement of the traditional 
questionnaire-based methods and to the integration of the varying points of view of design, 
support for learning and evaluation. This method that we call Integrated Pedagogigal Profile (IPP) 
can be seen as an advancement of the profile of pedagogical dimensions presented by Thomas 
I Reeves, but with some significant reconstructions. Our solution is based on the reconstruction of 

i the dimensions and the creation of a fuzzy picture that can give an overall sight into the many 

aspects of learning environment, yet not giving a false impression of preciseness. This dynamic 
picture which integrates the mentioned points of view in interdisciplinary fashion is achieved by 
gathering the individual results and representing these by circles the opacity of which is related to 
the size of the sample, thus forming a general picture of the essential features. 



Introduction 

The aim of this work is to present a method for description and evaluation of learning environments which pursues 
to the improvement of the traditional questionnaire-based methods and to the integration of the varying points of 
view of design, support for learning and evaluation. This method can be seen as an advancement of the profile of 
pedagogical dimensions presented by Reeves (Reeves 1997), but with some significant reconstructions. 

Currently the literature concerning constructivist approach to learning, learning environments or web- 
based learning environments entails various formulations for normative principles of learning, such as those of 
Jonassen, Salomon & Perkins or De Corte (Jonassen 1995;Salomon & Perkins 1996; De Corte 1995). What is 
lacking is the methodology for the evaluation of these principles. Usually the fundamental argument for the use of 
certain principles in the design of studies has referred to the purposefulness of the methods. However, the 
conceptual apparatus with which to evaluate what kind of learning environment fits the purpose in particular cases 
has been quite inadequate for practical work, because the understanding of the general concepts in such a level that 
enables accurate evaluation requires expertise in the field of educational science. In this work we will first discuss 
the problems of currently used methods and the theoretical grounds for our attempt to tackle these problems. This 
is followed by an in-depth example of our method. 



Current Problems in Design and Evaluation 

There are several reasons why the systematic evaluation of learning environments should be improved. Reeves lists 
four essential flaws in this area in his article “Evaluating What Really Matters in Computer-Based Education”. 1) 
Marketing the effectiveness of the technological innovations has surpassed the evaluation of their effectiveness 2) 
the quantitative evaluation of CBE has often been reduced to simplistic figures with extremely limited utility, such 
as the amount of money spent on hardware and software, the ratio of students to computers, or the amount of time 
students have access to CBE within a fixed period of time 3) the evaluations that have been previously conducted 
have usually been presented in the format of social science research reports which are not very accessible for wider 
audiences 4) the comparative studies have treated the instructional alternatives unproblematically as holistic 
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entities with meaningful differences (Reeves 1997). Hirsjarvi has expressed the weaknesses of questionnaires as a 
research method which concur with observations of Reeves. The studies based on questionnaires are considered 
typically superficial and theoretically modest. There are also some specific flaws, she spells out. First, there is no 
possibility to make sure how seriously the participants of the study have answered the questions. Second, it is not 
clear how apt the questions were from the point of view of the participants. The misunderstandings are hard to 
control. However, the questonnaires have real advantages, first and foremost their efficiency (Hirsjarvi 1997). 

The problematics in this field can be summarized in the following manner. Qualitative methods and their 
methods of representation (such as social science research reports Reeves mentioned) are informative, but time- 
consuming. On the other hand quantitative methods, such as questionnaires, are user-friendly but superficial. Thus, 
our task has been to develop methods that would be efficient enough to be used in everyday educational practice, 
yet improving the informative value compared to the Likert-style questionnaires. 



Theoretical Backgrounds 

The points of view that we have attempted to integrate into our method can be characterised by the 
different functions of the method. First, the method and the mode of representation have descriptive function 
describing the qualities of learning environment by using bipolar scales. This idea has its origins in Osgood’s 
method which is known as the 'semantic differential' (Osgood 1957). Reeves uses analoguous pedagogical 
dimensions in his profile (Reeves 1997), which we have attempted to develop further. Second, the method can 
be used as a Likert-style format of questioning and tries to capitalise its advantages while improving some of 
its defects. Third, the method can be used as a tool of explication in the design-process as well as in the 
learning-process. This function is derived from Donald Norman’s conception that intelligent activity is based 
on the external expression of thoughts (Norman 1993). 

One theoretical point of view inherent in our model arises from the criticism of the comparative 
method mentioned by Reeves. Either the instructional alternatives have been treated as cohesive entities which 
are opposed against each other, or these paradigms have been broken down to essential features which are 
similarly in opposition. These kinds of comparations are often presented in the context of justifying why the 
new paradigm should be adopted and the old abandoned. While these comparisons give some advice for 
evaluation of existing learning environments, they suffer from the values embedded in them -the new 
paradigm is seen as representing 'good' and the old paradigm as 'bad' or even 'evil'. This approach does have 
serious limitations and may also have serious practical consequences if applied carelessly. 

When this kind of good/bad- distinction is present in the evaluation, it also leads to a situation where 
the 'good' qualities are actually treated as necessary conditions for optimal learning. Either all these conditions 
are met or the learning environment is less than optimal. This approach also easily advocates a normative 
picture of a learner who is continuously active, self-reliant, reflective, intentional, immersed etc., whatever the 
qualities in the particular list happen to be regardless of whether this kind of human being could even exist. 
Constructivism is in threat of becoming vulnerable to the same critique it has presented to the earlier tradition 
if it presents one and only one ideal for a learner, when the purpose has been just the opposite -to support 
different kinds of learners as well as possible. Following from these problems one methodological commitment 
in our work has been to find a more neutral stance which would enable assessing the different instructional 
alternatives according to the purpose of the learning-activity. 

The problem of controlling misunderstandings in the Likert-style questionnaires that Hirsjarvi points 
out is partly caused by the singular nature of the assertions with which one is supposed to estimate one’s 
agreement. Ambiguity is an unavoidable feature of natural language which is epitomised when an expression 
is interpreted in isolation. However, even natural language is regular as the meaning of expressions is related 
to their context, or language-game in philosopher Ludwig Wittgenstein’s vocabulary. The more a person gets 
hints of the context of expression the more probable the correct interpretation becomes. (These issues have 
been previously discussed by the authors related to the problems of speech recognition (Hautakangas & Ranta 
2000).) Thus the developed method should provide improved means for interpretation, if this flaw is to be 
decreased. 

One characteristic problem in the design of web-based learning environments seems to be that one is 
attempting to create new practices at the same time with the tools for the practice with only limited help from 
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the general principles or ideals of learning. This means that the design process begins with an inevitably 
blurred vision of both the learning environment and its tools and the ways of learning that will emerge by 
using it. Thus, there is a need to support the explication and clarification of such vague idea. Norman says that 
the skill to use systematically the external modes of representing knowledge, such as writing, notes or 
visualization, is an important prerequisite for overcoming the limitations of intellectual activity. 

• It helps an individual to join information obtained from different perspectives and to improve the cohesion 
and consistence of the conceptions and explanations. 

• Writing and visualization of information is a process which contains many cognitive iterations. 

• It helps to surpass the limitations of person’s cognitive resources by releasing the capacities of memory into 
other use. 

• The significance, coherence and general meaning of the explicated information is easier to evaluate than the 
internal ideas of the mind. 

• Explication provides a kind of basis on to which one can build new or more structured thoughts and on 
which the creation of thoughts is easier. 

• External representation gives other people a good opportunity to engage in the collaborative development of 
the idea or thought and the building of new thoughts. (Norman 1993) 



Characteristics of the Integrated Pedagogical Profile 

As said earlier, our method is based on the Osgood’s semantic differential which utilizes the bipolar adjectives. 
More specifically we attempt to make an advancement to the profile of pedagogical dimensions presented by 
Reeves. In this section we will concentrato on discussing one of the basic ideas of Reeves and give a detailed 
account on how our method is generated from this idea by taking one dimension as an example. Reeves is breaking 
down the general instructional alternatives to 14 pedagogical dimensions in his article. These are presented in the 
opposing ends of a scale. There is an example of one of the dimensions in the picture below. Our approach is to 
consider this as a partial definition of the concept ‘teacher role’ which defines the applicable range of the concept 
by using dichotomical limits as the ends of the scale. 



Teacher Role 

◄ ► 

Didactic Facilitate 

Figure 1: An example of pedagogical dimension (Reeves 1997) 

However, even if the concepts by which ‘teacher role’ is defined are presented in non-negative terms, which 
is one of the traditional requirements for scientific definitions (Niiniluoto 1986), the presentation has defects. The 
concepts in differing ends do not represent genuine opposites in the context of learning, which they should if the 
scale is supposed to cover the whole range of the phenomenon which is to be evaluated. Thus, the first step in the 
generation of our method is to rethink what the concepts representing the oppositions should be. First, the concept 
‘facilitative’ does not represent an extreme case, which would rather be ‘none’, which is the case in pure self-study. 
The other end is a harder case. The concept should represent a situation where the teacher has a total control of the 
learner and his thoughts. One advocate for the concept would be ‘totalitarian’ which has also other connotations 
but for the time being it will suffice. After this we have a new scale. 

Teacher Role 



None 

Figure 2: A reconstructed scale for teacher role. 

To consider this as a research method, one does have a concept which can be used to pose a question (e.g. 
"What is teacher role in learning-environment x?”) and the scale with which teacher role can be positioned. This is 




Totalitarian 
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still a straightforward application of Osgood’s method which uses adjectives on the ends of the scale to capture the 
differences (Osgood 1957). However, the next logical step would be to apply these basic ideas by operationalising 
these concepts into empirical level. And one should notice, as Hirsjarvi points out, that the operational definitions 
are not used to specify the entire meaning, but the method for measuring (Hirsjarvi 1997). An example of the 
operationalisation into questionnaire -level could be: 

“I was able to make < > “The schedule of the course was fixed.” 

my own schedule for 
the course” 

Also what lacks from the account of Reeves is the dimension of time. As learning is a process, the methods 
of describing and evaluating learning in the learning environment have to include the dimension of time as well. 
This means that the appropriate amount of support that the teacher is giving in the learning process can and often 
does vary through time. This principle is traditionally described as ‘scaffolding’. One should notice that in the 
learning-process time can also be seen as cyclical. A learner enters the learning environment many times, he is 
each time a little more experienced in the goal-oriented activities he participates in. 

Now, what are the advantages that are received from this kind of method? 1) It allows one to describe the 
dimensions of learning environments more neutrally. It provides a support for evaluation of whether described 
qualities of learning environment are purposeful in a particular context and to particular goals, without making the 
scale itself biased. Thus, the instructional alternatives can be evaluated in different contexts. 2) The described 
structure gives more information to support a correct interpretation of the assertions compared to the Likert-style 
questions where the operationalisation has to lead to singular expressions. One can use the assertions in both ends 
of the scale and the information that they represent the opposites to interpret the assertion as well as the internal 
logical order of different questions. In addition, giving judgments is an activity that requires skill, and it may be 
anticipated that when the methods are first used the judgments of novices may differ greatly. 3) In the design- 
process the systematical use of dimensions pushes the designers to pass judgment on different aspects of the 
learning environment. The profile gives a common ground for the discussion in the design-team. As the reasons 
for individual judgments which can differ from designer to designer with different backgrounds are discussed 
thoroughly using the explicated profile as a point of reference a shared understanding of the characteristics and 
objectives of the learning environment is easier to reach. I.e. Nonakas & Takeuchi’s familiar idea of the tacit 
knowledge of the participants getting explicated (Nonaka & Takeuchi 1995) may be achieved through the use of 
the method. Thus, the design process becomes more transparent and structured, and the experience and created 
best practices can be used as new prototypes or guidelines in future work. 

The use of a systematic profile means also that different aspects of learning environment, such as usability 
(Nielsen 1993), content, creation of new learning-practices etc., can be focused on in controlled fashion which 
enables a better organization of the working-process. And as one reminds also that designing learning 
environments is structurally a project, focusing on the essentialities is as important as it is in research. This is 
highlighted in many works on project management and knowledge management by authors such as Goldratt or 
Davenport & Prusak (Goldratt 1997; Davenport & Prusak 1998). 

One should also note, that the tool itself is evolving. In our current project we have listed so far 49 concept- 
pairs which are preliminarily grouped using Jonassen's qualities of meaningful learning as general concepts or 
categories under which the multitude of different concept-pairs representing different points of view are structured. 
The use of this pattern brings out the dependencies and inconsistencies in this pattern, which, if treated as a 
picture, is inavoidably unsatisfying because the research itself involves the points of view of different disciplines. 
The evolution of this pattern should, however, bring out information about the essential features that should be 
taken into account in particular contexts. 



Possible Critiques and Future Prospects 

One obvious question that may arise from the critical reading of the described method is whether these 49 
or more bipolar dimensions could exhaustively cover the whole phenomenon of learning in such manner that by 
summing up the singular dimensions we could get a valid picture of the process of learning in its totality. Another 
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question is whether every single dimension should be regulated in order to reach an ultimate optimum of learning. 
The answer to these questions is no, and that there is no point in attempting to achieve such goals. As learning- 
process is tied to a particular context and situation one cannot hope to end up with general laws that could be 
applied in somewhat mechanical fashion. And it is not possible to evaluate whether there is a best practice of 
learning even if the context and objectives are known, rather, one can evaluate between better and worse practices 
that can be described and evaluated with support of other methods such as expert evaluation of the performance 
etc. Also, our goal is to find out what features turn out as most important factors in different contexts in order to 
guide the learning-processes to directions that have proved successful. 

Related to this Reeves faces critique by using a method of visualisation where he makes a profile of the 
dimensions of learning environment by linking the different values together with a line, even if the phenomenon is 
not continuous and the categories are not exclusive. Thus, the method of visualisation is also an important aspect 
of the methodology. Our solution is based on the creation of a fuzzy picture that can give an overall sight into the 
many aspects of learning environment, yet not giving a false impression of preciseness. This picture is achieved by 
gathering the results of the questionnaires and representing these by circles the opacity of which is related to the 
size of the sample. By combining the results into one picture one gets a rough but informative picture of all 
answers. From this picture one can find the emerged clusters of answers which are darker and also pick out the 
exceptional cases (if considered meaningful) that can be easily distinguished from the general case. 



Totalitarian" 



Teacher Role 




None 



Figure 3: An imaginary example of the method (n = 30). 

And as the results may be gathered as the time passes in the learning-process the visualisation can be presented by 
an animation which brings into focus the dynamic change in the results, i.e. the dynamic invariance which is a 
familiar concept already from the history of science (Routio 2000). This animation is analoguous to the 
visualisation used in the weather forecasts. 

Our challenge is to find out the valid scale which determines the opacity of an individual answer. This 
means that by research where we apply multiple methods we have to find the saturation point for the number of 
answers in order to set the darkest value of scale. Until that the visualisation can only be used as a relative scale 
which varies accoring to the number of answers. 
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